Adopt the module-max SDK interface from dagger/dagger#13992 - #20
Merged
eunomie merged 9 commits intoSep 8, 2026
Merged
Conversation
Design and implementation plan for replacing the beta SDK-module interface (initModule, targetRuntime, the @generate hook, and currentModule.asSDK) with detectScope and generateScope, as specified by dagger/dagger#13992. Signed-off-by: Yves Brissaud <yves@dagger.io>
dagger/dagger#13992 replaces the beta SDK-module interface with detectScope and generateScope, and removes CurrentModule.asSDK, ModuleSource.generateLocalDependencies, initModule, targetRuntime and the @generate hook. It ships no compatibility adapter, so every public function of this module selected something that no longer exists. detectScope answers with the directory of the nearest pom.xml. Every Java module has one at its root, and the SDK vendored under a module carries none of its own, so the nearest hit is always the owning module — python-sdk has to lift a hit out of its vendored sdk/, Java does not. generateScope scaffolds a scope that has no config from the template and writes its dagger-module.toml through the engine's manifest builder, then generates it; a scope that already has a config is only generated. A new module is generated even under a generate skip marker, which only holds a module that already exists. The builder has one runtime setter per builtin runtime, and this SDK targets github.com/dagger/java-sdk/runtime, so the runtime is named by loading a seed file: ModuleManifest.Validate accepts a non-builtin runtime on a manifest loaded from a config file and rejects it on one built from nothing. In a module scope the client set becomes the module's dependency set. The manifest's dependencies are cleared structurally, with the builder's withoutDependencies, rather than by name: withoutDependency matches an unnamed dependency on its source, and reading the recorded names means resolving every one of them, so a single stale entry would fail generation instead of being dropped. A manifest that already records exactly the requested clients is left alone, decided by comparing the builder's rendering of both sides, so a hand-written one is never reformatted for nothing. That cuts both ways — a module that records dependencies and has no clients loses them — so the fixtures pin it, in both manifest formats. Standalone clients, in a scope without a module, are refused, as python-sdk refuses them: every generated binding lives under a module's sdk/ and is compiled by that module's pom.xml, so there is nowhere to put one. Mod keeps its Maven machinery and gains `generated`, the workspace with the module's vendored SDK and entrypoint merged in. Its local-dependency staging goes: the engine orders scope generation by dependency itself. `generate`, `path` and `hasMarker` go with the interface they served. The SDK registers under [sdks.java] with one scope per fixture, and the e2e checks move to the new interface. They call this module, so on the released engine every one of them fails; [modules.e2e] check.skip covers them. sdk-sdk goes with the contract it validates: it asserts initModule, the as-sdk marker and `dagger module deps list`, none of which survive. The dang-sdk as-sdk block goes too, with no replacement: dang-sdk has not adopted the new interface yet (dagger/dang-sdk#13), and this repository's Dang modules have no generated files to produce. Signed-off-by: Yves Brissaud <yves@dagger.io>
The released engine has none of dagger/dagger#13992, so it cannot run the interface this SDK now implements: every call into the module fails with `"moduleManifest" not found`. The checks that exercise the interface therefore have nowhere to run on a released-engine `dagger check`. engine-e2e builds an engine from the branch, at the commit engine-dev is pinned to, and runs it as a playground with this checkout mounted inside. dev-sdk-check initializes a Java module through the CLI and calls it, which is the whole path: loading the SDK module, validating its function signatures against the engine's contract, generateScope, Maven, and the generated module serving its API. sdk-contract-check runs the gated e-2-e checks in the same engine as `dagger call` invocations, which `check.skip` does not filter. Both the engine-dev dependency and the engine source name the same commit, so CI does not float with a branch that force-pushes. Bumping the branch means bumping both, and dagger.lock with them. The dependency carries an explicit pin as well: without one, resolving it walks a local path that does not exist and dagger-dang-sdk:generate fails. Signed-off-by: Yves Brissaud <yves@dagger.io>
The engine no longer merges an SDK's init output with its own bookkeeping: it records a scope in dagger.toml and asks the SDK to produce the whole thing. So the commands change shape — `dagger module install`, `dagger module init java --name … --path …`, `dagger generate` — and `dagger call java-sdk init` / `generate --path` are gone with the functions behind them. Three things a reader now needs and could not find before: what makes a directory a Java scope (a pom.xml), that a module's clients are its whole dependency set, so an existing module's dependencies must be re-registered as clients before the first generate, and which engine each half of the check suite needs. The "codegen flag" section goes. It documented `init` writing codegen.automaticGitignore into a module's dagger.json; `init` is gone and nothing here writes that key. What was worth keeping — the generated files are committed, so the runtime skips codegen at load — the opening paragraph already says. Signed-off-by: Yves Brissaud <yves@dagger.io>
The branch moved from 78c241b6 to 7e6fc93c and renamed two things this SDK selects, so this is not a re-pin. detectScope is now findClientRoot, and its result is nullable: an SDK reports "no client root here" with null, not with the empty string. The engine reads the result through dagql.Nullable and treats an invalid one as absent, so returning "" would have named a root at the workspace root. The manifest builder's dependency verbs are now explicit about whose dependencies they are: withDependency, withoutDependency and withoutDependencies became withLegacyRuntimeDependency, withoutLegacyRuntimeDependency and withoutLegacyRuntimeDependencies, matching the LegacyRuntimeDependencies field they write. generateScope's signature and the [sdks.<name>.scopes] config shape are unchanged. dagger.lock is refreshed with `dagger update`: it carries the new engine's build closure, and it also picks up dang-sdk's current head, which had drifted past the recorded one. Signed-off-by: Yves Brissaud <yves@dagger.io>
Removing it was wrong. The reasoning was that dagger/dagger#13992 drops as-sdk, that dang-sdk has not adopted the replacement yet (dagger/dang-sdk#13), and that a table the engine silently ignores is worse than no table. The last step does not follow: the released engine does not ignore it, dang-sdk reads it, and without a registration dang-sdk's generator fails outright with "current module is not installed as an SDK in this workspace" rather than reporting an empty module set. That failure was invisible here until dagger.lock was refreshed. The recorded dang-sdk head predated the change, and a warm engine kept serving the old resolution, so the check passed locally and in a fresh clone while CI — which resolves the branch head — went red. The module-max engine ignores the table, so keeping it costs nothing there. It comes out when dang-sdk adopts the new interface and can be registered under [sdks.dang] instead. Signed-off-by: Yves Brissaud <yves@dagger.io>
The branch moved again, 7e6fc93c to 8fd9b22b, and took the module manifest builder out of the engine with it. `moduleManifest` is no longer a builtin: it lives in github.com/dagger/sdk-helpers, a Dang module this SDK now depends on and reaches as `sdkHelpers.moduleManifest(loadToml:)` / `(loadJson:)`. That module declares engineVersion v0.21.9, so depending on it does not stop this one loading on a released engine. Three behaviours change with it, following dagger/python-sdk#26. The generate skip marker is gone entirely — the setting, Mod.skipGenerate, the marker files and the fixtures that relied on them. generateScope always generates. generateScope always writes dagger-module.toml from the builder, and migrates a pre-1.0 module: the contents of its dagger.json move into the new manifest and the dagger.json is removed, so the two cannot disagree. The runtime the module already named is preserved, so a module on the engine's builtin java runtime stays there rather than being moved onto this repository's. Path.relativeTo replaces the hand-rolled relative-path arithmetic. It is on the released engine too, so it costs nothing to adopt. The e2e module is no longer installed in dagger.toml. An uninstalled module is never reached by a released-engine `dagger check`, which is a stronger gate than the check.skip list it replaces, and sdk-contract-check now runs `dagger -m .dagger/modules/e2e check` inside the playground rather than naming each check. Because generation is unconditional, a scope a check drives has to be a module Maven can build. The config-only stub fixtures go, along with the scope registrations that named them; the checks build their scopes from the template instead and share one module name where they only need a module, since the SDK installs its jars under a per-module Maven version. generate-scope-migrate-check is new and covers the migration above. sdk-helpers is pinned and locked. An unpinned dependency of this kind already cost this branch a red CI run once: the resolution is cached locally, so it keeps passing here long after it has moved upstream. Signed-off-by: Yves Brissaud <yves@dagger.io>
Picks up "Omit the default module source from dagger-module.toml". The builder's API is unchanged; only what it serializes moves, which the manifest assertions in the e2e checks cover. `dagger update` refreshes dagger.lock but leaves an explicit dependency pin alone, so the pin in dagger-module.toml and dagger.json is bumped by hand to the commit the lock now records. Letting the two disagree is how a generate check goes red in CI while passing locally. Signed-off-by: Yves Brissaud <yves@dagger.io>
`packager:generate` copied the whole io/dagger subtree out of the local Maven repository. That repository is a cache volume shared with module generation, which installs io/dagger/dagger-java-sdk/<module> and io/dagger/dagger-java-annotation-processor/<module> into it under a per-module version. So what landed under prebuilt/ depended on whether a generation had run against the volume first, and the check reported drift after a build that had succeeded. Copy the two published artifacts and the metadata file by name instead. The export no longer depends on what else is in the volume. The mount is LOCKED here as well, matching the SDK's own codegen containers: both write to that repository, and one volume takes one sharing mode. Locking alone does not fix this — it orders concurrent access, while the contamination is state left behind by an earlier writer — but seeding the repository from prebuilt/ is a plain recursive copy that two writers do race on. The failure needed a full `dagger check` to reproduce: `dagger call packager generate` on its own leaves the volume without the per-module artifacts and reports no changes. Signed-off-by: Yves Brissaud <yves@dagger.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dagger/dagger#13992 (branch
sdk-ux-module-max, pinned here at8fd9b22b5416f8dc7cb420ba37769adef6e874d2) changes how the engine drives an SDK module, with no compatibility adapter:CurrentModule.asSDKandModuleSource.generateLocalDependencies.JavaSdk.modulesselected the first andMod.generateModulethe second, so on that engine every call into this module failed.initModule,targetRuntime,@generate) withfindClientRoot(ws)andgenerateScope(ws, isModule, name, clients). The SDK is recorded under[sdks.java]indagger.toml, the engine setsWorkspace.cwdto a persisted scope, and the SDK writes the module manifest itself through the engine's manifest builder.This follows dagger/python-sdk#25 (merged), dagger/go-sdk#37 and dagger/dang-sdk#13, which adopt the same interface the same way.
Change
findClientRootanswers with the directory of the nearestpom.xml, relative to the workspace root, and null when there is none. Every Java module has one at its root, and the SDK vendored under a module carries none of its own, so the nearest hit is always the owning module — python-sdk has to lift a hit out of its vendoredsdk/, Java does not. A project built with anything but Maven has nopom.xml, so it is correctly not a scope.generateScopescaffolds a scope that has no config from the template, then generates it.generateScopewritesdagger-module.tomlthroughgithub.com/dagger/sdk-helpers, pinned at64645f1967d3dba6fce951dd61ae4acd8d9b0861. That module keeps the engine's rule that a non-builtin runtime is accepted on a manifest loaded from a config file, so this SDK can still namegithub.com/dagger/java-sdk/runtime.Mod.skipGenerate, the marker files, and the fixtures that relied on them.generateScopealways generates.generateScopealways writesdagger-module.toml, and migrates a pre-1.0 module: the contents of itsdagger.jsonmove into the new manifest and thedagger.jsonis removed, so the two cannot disagree. The runtime the module already named is preserved, so a module on the engine's builtinjavaruntime stays there.e2emodule is no longer installed indagger.toml. An uninstalled module is never reached by a released-enginedagger check, which is a stronger gate than thecheck.skiplist it replaces;engine-e-2-e:sdk-contract-checkrunsdagger -m .dagger/modules/e2e checkinside the playground.withoutLegacyRuntimeDependencies, not by name:withoutLegacyRuntimeDependencymatches an unnamed dependency on its source, and reading the recorded names means resolving every one of them, so a single stale entry would fail generation instead of being dropped. A manifest that already records exactly the requested clients is left alone, decided by comparing the builder's rendering of both sides, so a hand-written one is never reformatted for nothing.sdk/and is compiled by that module'spom.xml, so there is nowhere to put one.Modkeeps its Maven machinery and gainsgenerated, the workspace with the module's vendored SDK and entrypoint merged in. Its local-dependency staging is removed: the engine orders scope generation by dependency itself.generate,pathandhasMarkergo with the interface they served.dagger.tomlmoves to[sdks.java]with one scope per fixture.[modules.sdk-sdk]goes with the contract it validates — it assertsinitModule, theas-sdkmarker anddagger module deps list, none of which survive. The dang-sdkas-sdkblock goes too, with no replacement: dang-sdk has not adopted the new interface yet (Support new SDK interface (1.0.0-beta.12 and later) dang-sdk#13), and this repository's Dang modules have no generated files to produce.hack/designs/2026-09-04-sdk-module-interface.md.Verification
The released engine (
v1.0.0-beta.11) cannot run this interface. Dang infers a whole program on each call, so on that engine every call into this module fails — measured, along with the fact that the module still loads, which is what makes uninstalling thee2emodule sufficient. Thee2emodule is therefore uninstalled fromdagger.tomland its checks are replayed inside a development engine instead.engine-e-2-e:dev-sdk-checkbuilds an engine fromsdk-ux-module-maxat8fd9b22binside Dagger. There,dagger sdk listreportsjava;dagger module init java --name sdk-smoke --path .dagger/modules/sdk-smokewritesdagger-module.tomlnaminggithub.com/dagger/java-sdk/runtime(and nodagger.json),pom.xml, the module class,sdk/src/main/java/io/dagger/client/Dagger.javaand the generated entrypoint; anddagger -m … call container file --path /etc/alpine-release contentsruns the module.engine-e-2-e:sdk-contract-checkreplays the gatede-2-echecks inside that same engine, run asdagger -m .dagger/modules/e2e check:find-client-root-check,generate-scope-clients-check,generate-scope-init-check,generate-scope-migrate-check,nullable-return-check.generate-scope-migrate-checkis new and covers the pre-1.0dagger.jsonmigration. This is coverage python-sdk#25 does not have.generate-scope-clients-checkcovers both manifest formats — a client recorded indagger-module.tomland in a pre-1.0dagger.json, dropped again in both — asserts that a module with dependencies and no clients loses them, and generates a new module with a client to prove the client's type reaches the vendored bindings.packager:unit-tests,packager:generate,templates:generateanddagger-dang-sdk:generatepass on the released engine.Both the
engine-devdependency and the engine source name the same commit, so CI does not float with a branch that force-pushes; bump both, anddagger.lockwith them.Known limitations
v1.0.0-beta.11,dagger call java-sdk find-client-root --ws .fails withfield "withFile" not found in Dagger.Workspace—generateScopewrites the manifest throughWorkspace.withFileand removes a migrateddagger.jsonthroughWorkspace.withoutFile, and that engine has neither.is-moduleandnameon a scope — it never seedsclientsfrom an existing dependency list. Each dependency must be re-registered as a client first. The README says so.dagger module client adddoes not persist onsdk-ux-module-maxat8fd9b22b. It loses the workspace overlay on reload and silently writes nothing, on every SDK; the fault is in the CLI (internal/cmd/dagger/module_sdk.go), not in any SDK'sgenerateScope. Client handling is therefore verified at the API level, callinggenerateScopefrom Dang and diffing the result — the same way python-sdk#25's checks avoid it.dependencySource'sGIT_SOURCEarm needs a real remote no check here can produce hermetically, and no check runs a full generation of a pre-1.0dagger.jsonmodule — its manifest branch is covered, the Maven half is not.detectScope→findClientRoot(now nullable) and the manifest builder's dependency verbs. Bothengine-e2epins and the docs name one commit, so a later move is a visible bump rather than silent drift.mod.dangsurfaced once the e2e checks began running in parallel: the shared Maven cache volume is seeded with a plain recursive copy, and two concurrent generations collide on it. The/root/.m2mount is nowLOCKED, so the mount serializes instead of the callers.